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Abstract 

Recent innovations in the design of computer viruses have led to new trade-offs for the attacker. 
Multiple variants of a malware may spread at different rates and have different levels of visibility to the 
network. In this work we examine the optimal strategies for the attacker so as to trade off the extent 
of spread of the malware against the need for stealth. We show that in the mean-field deterministic 
regime, this spread-stealth trade-off is optimized by computationally simple single-threshold policies. 
Specifically, we show that only one variant of the malware is spread by the attacker at each time, as 
there exists a time up to which the attacker prioritizes maximizing the spread of the malware, and after 
which she prioritizes stealth. 


Index Terms 

visibility, optimal contagion, malware epidemics. 

I. Introduction 

Malware (i.e., viruses, worms, trojans, etc.) has been a prominent feature of computer networks 
since the 1980’s [1], and has evolved with the growing capabilities of computing technology. 
Anderson et al. [2] estimated that malware caused $370m of damage globally in 2010 alone. 
Traditionally, malware was designed with the express aim of infecting as many machines as 
possible, leading to the mass epidemics of the early 2000’s (e.g.. Blaster [3]). More recently, the 
focus has shifted to more “surgical” strikes where visibility is highly undesirable, as awareness 
can lead the intended target to cease communication (e.g., by quarantining the targets). The 
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malware Regin was only discovered (in 2014) after operating since at least 2008, and was so 
complex that even when its presence was detected, it was not possible to ascertain what it was 
doing and what it was targeting [4]. Stuxnet, as another example, was designed to attack a 
specific control software used in centrifuges [5] and did not steal or manipulate data, or receive 
any command instructions from remote sources so as to maintain stealth [6]. Furthermore, its 
very presence in a system was undetectable due to a rootkit [5]. Yet, it was discovered and 
remedied after it spread outside its target area [7] (cf. Duqu, Flame, and Gauss [8]). Thus there 
is a new critical trade-off for the attacker — to ensure maximum damage while minimizing 
visibility to the defender. 

We now describe different dimensions of this trade-off. Malware spreads from one computing 
device to another when there is a communication opportunity between the devices. In networks, 
both wired and wireless, inter-node communication can be visible to the network administrator, 
and can serve as a way of detecting the presence of malware before its function is fully 
understood. However, the attacker also has a conflicting onus to ensure the rapid propagation of 
her program, as computer systems evolve at a rapid pace, and the exploit(s) that the malware 
targets will be noticed and patched in due course. Furthermore, some malware designers work 
to specific deadlines — e.g., Stuxnet was due to become inoperational in June 2012 [9]. On the 
other hand, the second variant of Stuxnet was released to spread faster (and thus in a more risky 
manner) after the designers were concerned about its limited spread [6]. Thus, an attacker will 
seek to minimize her communication footprint while still trying to ensure the timely spread of 
the malware. 

In particular, we consider the case where two variants of a single emerging malware spread in 
a network that is unaware of their existence. One spreads aggressively in every contact, and is 
thus visible to the network due to its communications, while the other, passive, variant does not 
spread subsequent to infecting a node. We assume that the network cannot determine the infection 
state of any particular node and does not have patches to remedy the attack, but can detect an 
attack by looking at the unusual communication patterns (e.g., the transfer of malware between 
nodes) resulting from the malware attack. Coordinating distributed attacks comes at the cost of 
added visibility due to communication and is susceptible to timing errors in the hosts. Thus, we 
focus on the case where distributed nodes that are infected are not asked to coordinate, as was 
the case in Regin and Stuxnet. The natural question that arises is to characterize the structure 
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of the optimal malware variant mix that the attaeker will spread at eaeh instant depending on 
their goal struetures and the eommunieation meehanisms that they may have at their disposal. 
This is an imperative first step to devising remedies for sueh attaeks. 

A. Problem Description 

We eonsider a network under attaek by these two variants of a malware. Depending on their 
infeetion status, nodes ean be divided into 4 groups:^ Germinators (G), Suseeptibles (S), Zombies 
(Z), and Passives (P). We now deseribe these states, as well as their dynamies and the impaet 
of the attaeker’s eontrol (as will be elueidated in §III.A). We also outline an augmentation to 
the model that is eonsidered in §III.B and adds a further possible meehanism of interaetion and 
eontrol to the dynamies: 

1) Germinators (G): 

- are di fixed (potentially very small) fraetion of nodes, - are the only nodes under the attaeker’s 
direet eontrol, 

- are the only nodes that ean choose how to internet with suseeptibles and zombies depending 
on the goal of the attacker: at each encounter with a susceptible, they decide whether to turn it 
into a zombie or a passive, or to leave it as a susceptible. 

- damage the network by executing malicious code, 

- are visible to the network due to their communications. 

- in an augmentation in §III.B, we add a further mechanism of interaction (halting) whereby 
the germinators, upon contact with zombies, can turn them into passives (i.e., stopping them from 
spreading the message any further). This can potentially lead to the attacker initially utilizing 
epidemic spreading and then halting the spread once the marginal benefit of infection is overtaken 
by the marginal effect of visibility, leading to to a potentially longer propagation of the zombies. 

2) Suseeptibles (S): 

- are nodes that have not received any variant of the malware, 

- upon receipt of the malware from germinators, they can turn into zombies (Z) or passives 

(PI 


*Note that this classification and the resulting dynamics are an abstraction of real world networks and sacrifice some accuracy 
for modeling simplicity. However, these assumptions are common in cybersecurity literature, e.g., [1], [10] and lead to significant 
insight. 
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- upon receipt of the malware from zombies, they will turn into zombies (Z). 

3) Zombies (Z): 

- have received the aggressive malware variant, 

- damage the network by executing malicious code, 

- will continue to propagate the aggressive variant indiscriminately (i.e., upon meeting a 
susceptible, will turn into a zombie), 

- are visible to the network due to their communications. 

- in the augmentation in §IILB, the additional mechanism of halting can turn zombies into 
passives. 

4) Passives (P): 

- have received the passive variant of the malware, 

- damage the network by executing malicious code, 

- will not propagate the malware variant any further, 

- contrary to germinators and zombies, are invisible to the network as they do not communicate 
with other nodes to spread the malware henceforth. 

These states and their properties are summarized in Table I. We assume that all nodes mix 
homogeneously (i.e., contacts between nodes are independent and exponentially distributed) with 
rates that only depends on the infection states of the two nodes. Thus, all nodes that are in one 
infection state can be assumed to be identical from the perspective of the malware. The purpose 
of this abstraction is to simplify the interaction model for analysis in the population limit (i.e., 
as the number of nodes increases). 

In these models, the attacker controls the mixture of zombie and passive malware variants 
through the germinators under its direct control. Whenever a germinator meets a susceptible, 
based on the control chosen by the attacker, it spreads either the zombie or passive variant of 
the malware to the susceptible, or leaves it as it is. In the dynamics in §IV.B, the germinator has 
an additional controlled mechanism of action, whereby upon meeting a node with the zombie 
variant of the malware, it can replace the variant with the passive one (a “halting” mechanism). 
These controls are assumed to be piecewise continuous, but they can take any value between 
zero and one, which determines the percentage of relevant interactions for which the specified 
action happens. We do not assume that all nodes make the same spreading decision at each time 
instance: the attacker can assign a certain uniformly distributed and possibly varying fraction of 
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State 

Visibility 

Growth over time 

Propagation 

S 

N 

Only decrease 

- 

G 

Y 

Fixed 

Y 

Z 

Y 

Increase or decrease 

Y 

P 

N 

Only increase 

N 


TABLE I: The states of the SGZP model and their characteristics. “Visibility” denotes whether 
the infection state of the node is detectable by the network defender. “Growth over time” 
determines the possible changes in the fraction of nodes in each state over time (note that the only 
case in which zombies can decrease is the dynamics outlined in §IV.B). Finally, “Propagation” 
determines whether a node in that state can spread the malware to a susceptible node upon 
contact. 


germinators to make the same decision at each time, or it could allow all agents to make one 
of the two decisions with a certain, possibly varying probability at each time. The outcome of 
both cases is that a certain uniformly distributed percentage of interactions (derived from the 
attacker’s controls) lead to the creation of zombies and passives, and the rest have no effect on 
the potential target. 

Later, we also investigate the effect of defense strategies on the optimal spread of malware 
variants (§III.C). In these defense strategies, the defender limits the effective contacts of nodes 
using a pre-determined function of malware visibility (which changes over time) as a means to 
limit the spread of malware. We consider two classes of network defense functions: affine and 
sigmoid. These defense strategies, however, come at the cost of stopping legitimate communi¬ 
cation within the network. This is akin to choosing the communication ranges of nodes as a 
decreasing function of the visibility of the malware, which is a form of quarantine. 

We allow the attacker to choose the malware spreading controls so as to maximize a measure 
of overall damage (described in §III.E). We first consider a damage function that depends on 

a) malware efficacy, which is a function of the aggregate number of zombies and passives, and 

b) malware visibility, which is a function of the number of zombies (for the models in §III.A 
and §III.B). Then, we consider a damage function where malware efficacy is the attacker’s only 
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direct concern, and is thus the damage function to be maximized, for the case where visibility 
is built into the network dynamics through a network defense policy which is a function of the 
fraction of zombies (as in the model in §IILC). These formulations, to the best of our knowledge, 
have no precedent in the epidemics literature, and can be used to further investigate the effects 
of malware visibility in networks. 

An advantageous feature of all these models is that the malware designer only requires 
synchronized actions from a fixed number of nodes that are under its control from the outset. 
This decreases the risks of detection and policy implementation errors arising from coordinating 
synchronized distributed actions among a varying set of nodes. 

B. Results 

We then derive necessary structures for optimal solutions for each of the cases, using Pontrya- 
gin’s Maximum Principle and custom arguments constructed for each case (in §IV). We show 
that the attacker’s optimal strategy in all of these models is for the germinators to spread only 
one variant of the epidemic at each time: the germinators will create zombies up to a certain 
threshold time, and then only create passives (including by halting zombies) from then on. That 
is, the optimal controls are bang-bang (i.e., only taking their minimal and maximum values) 
with only one jump. Note that the controls can take any value between 0 and 1 at each point 
in time, and this bang-bang structure is one that emerges from the dynamics of the problem. 
These structural results are without precedent in the literature, both due to the uniqueness of the 
model, as well as the constraints placed on the vector of optimal controls. 

It is interesting to note that in each of the variations we consider, our analysis reveals that all 
the controls in each model have the same threshold, a fact that is not at all clear a priori. Thus 
the entire control space can be described by one time threshold. This structure is invaluable for 
deriving the optimal controls computationally (by solving the scalar optimization problem with 
the state ODEs mapping the variable to the damage objective). Furthermore, the controls are 
deterministic and easy to implement as the germinators need to be programmed with just one 
time instant for all of their controls. 

Finally, we investigate the performance of the derived optimal controls using numerical sim¬ 
ulations (in §V). We first investigate the effect of the additional halting action on the optimal 
attack policies. We show that for both the simple and halting models, as the rate of contact 
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between zombies and susceptibles increases, zombies are created for a shorter time period. We 
also show that the halting control adds to the length of time the zombie variant should optimally 
be propagated, with the additional propagation time depending on some system parameters. 
We then compare the optimal control with heuristics, and show that even without the halting 
control, the optimal solution performs 10% better than the leading heuristic, with the performance 
differential being larger for more naive heuristics. We then consider errors in the implementation 
of the network defense strategy outlined in §III.C, and investigate their effects on the malware 
spread. We show that erroneous estimations on the part of the defender only slightly affect the 
damage inflicted by the attacker, which points towards the robustness of the attack policies to 
errors in estimations by the network defense. Finally, we quantify the effect of synchronization 
errors among the relatively small number of germinators on the efficacy of the malware attack. 
We show that any such attack is robust to small errors among the germinators, sounding an 
alarm to the fact that these malware attacks are less vulnerable to implementation issues that 
may arise from synchronization errors than previous generations of malware. 

II. Literature Review 

Multiple interacting epidemics that spread among a single population have been considered 
in the fields of biology (e.g., multiple strains of a viral epidemic [11], [12]) and sociology (e.g., 
competition among memes in a world with limited attention span [13]). The key distinction 
between the control of biological epidemics [14]-[18] and that of malware ones is that in malware 
epidemics the attacker can also decide to use her resources optimally and to adapt to foresee 
the response of the defender. In the realm of sociology, the control of information epidemics 
offers closer parallels to that of malware. For example, Kandhway and Kuri [19] model how 
an erroneous rumor may be optimally stifled by the spread of correct information, which is 
a secondary epidemic that interacts with the naturally occurring rumor epidemic. However, in 
this case only one of the epidemics can be controlled, while the malware attacker can possibly 
simultaneously control the spread of all malware variants. When there are multiple controllable 
epidemics, the resulting simultaneous controls are interdependent, and focusing on one control 
and characterizing its structure does not lead to a characterization of the optimal action. Thus, 
in malware epidemics there are vectors of controls available to the attacker, which requires new 
approaches and techniques compared to the other fields discussed. 


7 



Even within the majority of malware epidemic models, e.g., [20]-[26], the spread of only 
one malware has been examined, while we focus on the case where two variants are spreading 
in conjunction with each other. This presents a fundamentally different choice to the attacker, 
and so the model presented for the spread of visibility-heterogeneous malware variants has no 
precedent in literature. Accordingly, the questions we asked and the solutions we obtained are 
substantially different to prior work. 

Note also that in nearly all malware epidemics, as well as the more generic epidemic models 
mentioned, some form of the homogeneous mixing assumption is used to obtain tractable results. 
While [27] provides one interesting avenue for the relaxation of the mean-field assumption in the 
study of a given epidemic process, tractable results in the epidemic control domain still critically 
rely on the mean-field assumption. 

Nonetheless, we still distinguish other aspects of our work from those considering a single 
type of malware: in these papers: 1- it is assumed that the attacker’s sole aim is to maximize the 
spread of the malware, which is no longer the case for the emerging class of surgical malware 
such as Regin [4] and Stuxnet [5] and 2- attackers have a mechanism to control the spread of 
the malware remotely in the future, e.g., through a timer in the code which would be executed in 
infected machines (as in [28]). Any such code would have to interact with the operating system 
of the infected node, the configuration of which might not be known to the attacker, and can 
thus create a point of failure for the malware. The failure of such a mechanism of control was 
key to the overspread and subsequent remedy of Stuxnet [7]. 

Among the work on the control of a single-type/variant of malware (and the closely related 
literature on the spread of a message in Delay Tolerant Networks [29], [30] and the spread of a 
rumor [19]), the closest work to this topic (in terms of approach and spreading models) was in 
two papers [26], [31]. In both papers, however, the authors assume that the malware can control 
the transmission range of infected nodes^ and patching is the major defense of the network 


^We assume that the control affects the mix of malware variants and that the communication ranges of nodes are outside the 
malware’s control, perhaps even being controlled by the defender as a mitigation mechanism. Thus, the control and the trade-off 
to the malware designer is fundamentally different. 
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and starts as soon as the epidemic spreads^. Thus, while the derived bang-bang structure of 
the optimal controls is similar, their models and their results apply to a fundamentally different 
class of malware, and the arguments used in deriving the results are only similar at the level of 
using a classic Maximum Principle-derived switching function argument for constrained controls. 
Furthermore, the adaptive defense model and the results on the simultaneity of 3 optimal control 
switching times for the halting model are without precedent in the literature. 

Finally, the very strict structure we prove for the vector of malware optimal control, which 
restricts the search space for computational methods to a single parameter, is also without 
precedent in any of the aforementioned literature. 

III. System Model and Objective Formulation 

In this section we model the spread of malware in a homogeneous network with random 
contacts. This can be the case where malware spreads among mobile devices with proximity- 
based communication, or where random contacts in an address-book are utilized. The virus 
propagates in the network between times 0 and T. We represent the fraction of susceptible, 
germinator, zombie, and passive nodes at time t with S{t), G{t), Z{t), and P{t) respectively, 
and assume that they are differentiable functions of time. We assume that for any pair of states, 
the statistics of meeting times between all pairs of nodes of those two states are identical and 
exponentially distributed, where the mean is equal to the homogeneous mixing rate of those two 
states. Groenevelt et al. [33] have shown that homogeneous mixing holds under the common 
Random Way-point and Random Direction mobility models (when the communication range 
of the fast-moving nodes is small compared to the total region). It has been shown [34], [35, 
p.I] that the resulting evolution of such a set of state fractions (where state transitions occur 
according to a Poisson contact process) will converge pathwise to the solution of a set of ordinary 
differential equations derived from the dynamics in the population limit (i.e., in the mean-field) 
on any limited time period (in particular, including the transient phase). In previous work, we 
have shown that such approximations are reasonable even with populations as small as 40-160 

^This may not be the case for an emerging stealthy epidemic like Stuxnet that is very large and extremely hard to decipher, 
let alone mitigate [9], [32]. In our model, the network only becomes aware of the malware as it becomes more visible (i.e., as 
the visible variant spreads). 
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Fig. 1: The blocks represent the 4 states of nodes with regard to the malware. The solid black 
lines show the dynamics in §III.A with the transition rates super-imposed. The green arrows 
point from each source of malware to the resulting transition. The dotted red lines show the 
additional halting action in §III.B. The model in §III.C has the same dynamics as the solid 
black lines, but with (3 being a function of Z (i.e., (3{Z)). 


[29] 3 

Note that the zombies can be programmed to only spread the malware at a fraction of the 
times they meet susceptibles, slowing their spread, or they can be programmed to use resources 
that are not utilized by the rest of the network to spread faster. Therefore we take the mixing 
rate between Z and S to be potentially different from the other pairs of states. 

We describe the state dynamics of such systems as an epidemic for the cases where: 1) 
germinator agents can only interact with susceptible agents (§III.A), 2) germinator agents can 
also interact with zombies as well (§III.B), and 3) effective network contact rates are a function 
of the infection spread, mirroring the response of a network defender (§III.C) (Figure 1). We 
state and prove a key observation about all these dynamics (§III.D). We next formulate the 
aggregate damage of attack efficacy and the ensuing visibility (§III.E). Finally, we lay out the 
optimization problem in §III.F. 

A. SGZP Model with no halting 

The attacker can spread the malware in two ways: 1- upon encountering a susceptible, she can, 
through the control variable uz{t), turn that susceptible node into a zombie, i.e., one that will 


''This work [29] also lays out a roadmap on how to partially relax the homogeneous mixing assumption in the current problem. 
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henceforth propagate that infection to susceptibles it meets. 2 - upon encountering a susceptible, 
she can, through the control variable up{t), turn that susceptible into a Passive, P. These control 
variables — {uz, up) G W, where U is the set of piecewise continuous controls — can be thought 
of as the probabilities that an interaction of a germinator and a susceptible at time t will lead to 
the susceptible becoming a zombie and a passive respectively. To maintain such a probabilistic 
intuition, we constrain their sum to be less than one. 


S = -/3GS{up + Uz) - iPZS 
Z = l3GSuz + ifiZS 
P = fiGSup 

Up P Uz 

0 <Mp<l 0 <MZ <1 


(la) 

(lb) 

(l c) 

( 2 a) 

(2b) 


Here, f3 is the mixing rate between S and G (which the attacker can calculate using time averages 
of contact times), and 7 /) is the mixing rate between Z and S (with 7 > 0). Thus, 7 is the 
relative secondary rate of spread of the malware. We consider all values of the parameter 7 , with 
an associated trade-off: if 7 is high, the zombies spread too fast and increase visibility, while if 
7 is low, the malware does not spread to cause significant damage. 

B. SGZP Model with halting 

This model is akin to the previous one, with one more mechanism added: germinator nodes 
(G) can force a zombie (Z) to become passive (P) through a process we will call “halting”. 
This happens through another control variable Uh, which, in keeping with the intuition, can be 
thought of as the probability of halting encountered zombies at each instant. Again, we take 
{uz,up,Uh) G W, where W is the set of piecewise continuous controls. The system dynamics 
become: 

S = —j3GS(up + Uz) — 'y/SZS (3a) 

Z = /3GSuz + 'y/3ZS-7r/3GZuh (3b) 

P = l3GSup+TTl3GZuh, (3c) 

with 0 < TT < 1 signifying the extent to which the zombies can be stopped when encountered by 
the original germinators. This model is similar to the Daley-Kendall rumor model [36], where 
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repeated interaction with active agents can turn an active spreader of the rumor into an agent 
that is aware of the rumor, but has no interest in spreading it any further. The constraints now 
become: 

Up + uz <l (4a) 

0 < Mp < 1, 0 < < 1, 0<Uh<l. (4b) 

C. SGZP Model with no halting and adaptive defense 

Instead of allowing a constant rate of interactions (3, the network defender can choose the 
effective mixing rate /3 to be a function of the fraction of zombies as her defense policy (/3(Z)). 
In these policies, the network defender regulates the rate of contact between nodes based on the 
proportion of zombie nodes it has observed. While the network cannot determine which nodes 
have been compromised, it can determine the fraction of the network that has been infected by 
zombies by observing the chatter among nodes and the extra communications whose purpose 
is unknown, either in the whole network or among a representative subset of nodes. If these 
illicit communications are significant enough to attract the network defender’s attention, they can 
implement a quarantine defense policy, captured by I3{Z), which will be a function of likelihood 
the malware is detected, and which will decrease the spread of the malware. 

We consider the system dynamics described in the no-halting model, and adapt them accord¬ 
ingly: 


^ = -/3{Z)GS{up + Uz) - ll3{Z)ZS 

(5a) 

Z = l3{Z)GSuz + ll3{Z)ZS 

(5b) 

P = l3{Z)GSup 

(5c) 


The controls available are also the same as those in (2). In particular, they are still assumed 
to be piecewise continuity. 

We consider two classes of 13{Z) functions: 1) Affine functions, of the form I3{Z) = —aZ + 
l3max for 0 < a < l3max (a natural assumption, as the contact rate cannot be negative). If 
a = 0, the affine case simplifies to the constant [3 case. 2) Exponential sigmoids, of the form 
/3z = ^ ^ a{z-z fe) ’ ^ ^ ^ ^ being a fixed threshold and a > 0 denoting the sharpness 
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of the cut-off. As a increases, I3{Z) can become arbitrarily close to I3{Z) = (3o'^z<Zth^ 
or-nothing policy. Both of these classes satisfy /3(Z) > 0 for all Z (i.e., the network never 
shuts down completely due to the infection) and < 0 for all Z (except for the trivial case 
of constant l3{Z)), as more visibility should lead to more communication restrictions from the 
network. In mobile epidemics, this is equivalent to nodes decreasing their communication range 
upon the detection of an infection, e.g. as in [37]. In practice, the network will have an estimate 
Z of the fraction of zombies. Our simulations reveal that the sub-optimality induced by the 
estimation error is small (§V). 

D. Key observations 

We start with a theorem that holds for all the models presented above, and which will be used 
as a building block to obtain structural results in §IV. 

Theorem 1. For a system with the mechanics described in either §III.A, or §III.C, with 

initial conditions S'(O) = 5*0 > 0, G(0) = Go > 0, Z{0) = Zq > 0, and P(0) = Po > 0, and 

So + Go + Zq + Pq = 1, and with piecewise continuous controls up, uz (and in (3), Uh), the 

dynamical systems (1), (3), and (5) have unique state solutions (S(t),G(t), Z(t), P(t)), with 
S{t) > 0, Z{t) > 0, P{t) > 0, and (S + G + Z + P){t) = 1 for all t G [0,T]. 

The assumptions S'o > 0 and Go > 0 are natural, otherwise there is no interaction to control. 

Henceforth, we will assume these, as well as Zq > 0 and Po > 0. 

Proof: The uniqueness follows from standard results in the theory of ordinary differential 
equations [38, Theorem A. 8 , p. 419] given the observation that the RHS of the dynamic systems 
is comprised of quadratic forms and is thus Lipschitz over [0, T] x S, where S is the set of states 
such that the boundary conditions hold. 

We provide the proof for the case of §III.A, and note the changes for §III.B. First of all, 
(S + Z + P)(t) = 0 and (5 + Z + P)(0) = 1 - Go, so (^ + G + Z + P)(f) = 1 for all 
t. We know that S = —/3GS(up + uz) — yPZS > —MS, where M is the upperbound of 
I3G + yPZ (because (up + uz) < 1). Therefore, S(f) > Soe~^^ > 0 for all t. Therefore, 
Z = (3GSuz + yPZS > y^ZS > MZ, where M is a lowerbound on 7 / 35 ' which exists due 
to continuity (respectively, Z = jdGSuz + yfiZS — Tr(3ZGuh > Z(yl3S — fPiGuh) > M'Z, 
where M' is a lowerbound on (y(3S — (IPrGuh) which again exists due to continuity). Note 
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that the first inequality resulted from uz{t) > 0 for all t. Therefore, Z{t) > > 0 

(respectively Z{t) > Z^e^'^ > 0) for all t. Finally, P = /SGSup > 0 for all t (respectively, 
P = (3GSup + 7rf3ZGuh > 0 for all t), as uz{t) > 0, so Pq > 0 leads to P{t) > 0 for all t. 

Theorem 1 can be proved very similarly for the model in §IILC using the reasoning we used 
for the model in §IILA, with the difference that in the arguments, /3 is replaced by I3{Z), which 
is lower-bounded away from zero for positive Z. ■ 

E. Utility Function 

As we discussed, the attacker tries to maximize attack efficacy while minimizing visibility. 
We capture efficacy as a function /(■) of the aggregate number of zombies {Z) and passives 
(P) at each time instant. Meanwhile, visibility is only a function of zombies that re-spread the 
malware, as that is the only time the malware is detectable. Visibility increases the likelihood that 
the network defender detects the malware and takes defensive actions. This means that we can 
capture instantaneous visibility as a function g{-) of the number of zombies at that instant. While 
the attacker cannot in general measure the malware’s visibility, she can choose g{-) based on how 
detrimental detection would be for her purposes. This formulation is comprehensive because the 
fixed number of germinators (G) both cause damage and are visible, and are implicitly a term 
that is added to the variable of both functions. This leads to the following aggregate damage 
function that the attacker seeks to maximize: 

J = r{f{Z{t) + P{t)) - g{Zm dt. (6) 

Jo 

We have some natural assumptions on /(.) and g{.): /(O) = 5f(0) = 0, with > 0 and 

df(Z+P) _ df{Z+P) ^ 
dZ dP ^ 

We assume that f{x) is concave, which means that incremental damage does not increase as 
the number of infected agents increases [i.e., the pay-off per infected agent decreases]. 

In §IV.A: We assume g{x) is convex. This means that an increment in the zombies is costlier 
(results in more visibility) when the infection is already more visible. This could be the case 
when the network becomes more wary of the infection as it progresses and becomes more visible. 
In §IV.B: We simplify g to be linear, ^((a;) = kgX, kg > 0. 

In §IVC: We set g{x) = 0, as the effects of visibility have been built into the network dynamics 
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through I3{Z). This leaves us with: 


J 


f{Z{t) + P{t))dt. 


(7) 


F. Problem statement 

In §IV.A and §IV.C, the attacker seeks to choose controls {uz,up) G U satisfying (2) so as 
to maximize J (respectively, (6) and (7)j, while in §IV.B, she seeks to maximize J (6) through 
a choice of (uz,up,Uh) G U' that satisfies (4). 


IV. Structural Results 


Using Pontryagin’s Maximum Principle and custom arguments specific to each case, we obtain 
the one-jump bang-bang structure of the optimal controls for the various cases in §IILA, §IILB, 
and §IILC. We provide the proof for §IV.A in the main text (§IV.D) and the ones for §IV.B and 
§IV.C in the appendices (§ Appendix A and § Appendix B respectively). 

Intuition is unclear in determining these structures: while intuitively creating zombies at the 
beginning of the time period allows the malware to benefit from their epidemic spread, it also 
penalizes the malware more because of its prolonged visibility. This is further complicated by the 
fact that the controls can take any value between 0 and 1, and thus it is possible for the attacker 
to have any mix of malware spread at each instance in time. The strict structures that arise from 
the analysis are counter-intuitive and interesting both theoretically and from an implementation 
standpoint. 


A. Results for the no halting model (proved in §IV.D) 
Theorem 2. Any optimal control in U will satisfy 


Up(f) 


for some t* G [0, T). 


{ 0 te[o,t*) 
1 te(t*,T) 


uzit) 


If tG[o,r) 
[o te(t*,T) 


This result means that for any optimal control, there exists a time threshold t* such that prior 
to t*, the germinators convert all the susceptibles they encounter to zombies, and subsequent to 
it they convert the susceptibles to passives. 
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The fact that creating zombies starts from the initial time for all interactions, that passives are 
created for a time period leading up to the terminal time for all interactions, and that the switch 
between creating zombies and passives is instantaneous - with no gap between, and no over-lap 
in, the intervals in which these variants are propagated, as well as no intermediate propagation 
rates - is not at all a priori obvious. 

Note that we prove a necessary condition for any optimal control, thus reducing the search 
space of controls from a vector of functions to a scalar (t*). This is a cause for concern, as 
the latter is much more computationally tractable for the attacker, and shows that any optimal 
policy will also be simple for the attacker to execute. The attacker can execute the optimal policy 
by optimizing the ODE (1), just varying the scalar parameter t*, and then coding t* into the 
germinators, which are the only nodes that execute the control. 

B. Results for the halting model (proved in §Appendix A) 

Theorem 3. Any optimal control in U' will satisfy 

{ ote[o,r) ite[o,r) 

Mt) = < 

ite(t*,T) [ote(r,T) 

for some t* G [0,T), except in the case where Z(t) = Q for all t G [0,T], in which case Uh can 
be arbitrary with the other two structures holding. 

This means that there exists a time threshold t* such that prior to t*, the germinators again 
convert all the susceptibles they encounter to zombies while not halting any zombies they meet, 
and subsequent to it they convert both the susceptibles and zombies they encounter to passives. 
Here, the added halting control can be used to slow the spread of zombies. 

The fact that the same result as Theorem 2 holds for uz and up in the presence of Uh is not 
clear a priori. Furthermore, the fact that the halting optimal control is bang-bang and that the 
switching time is the same as the other controls is surprising. 

C. Results for the adaptive defense model 

Theorem 2 holds (with the difference that t* G [0,T]) for constant, affine, and sigmoid ffiZ). 
This is remarkable given that here, (3 changes as a function of Z. This result is proved in 
§ Appendix B. 
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D. Proof of Theorem 2 for the no halting model 

Proof: This proof utilizes the neeessary eonditions for an optimal eontrol derived from 
Pontryagin’s maximum prineiple. In partieular, we explieitly eharaeterize the optimal eontrols 
as funetions of the optimal states and co-states (akin to Lagrange multipliers). Subsequently, we 
start at terminal time, where the eo-states are known, and follow their evolution baekward in 
time till we arrive at the initial time, thereby implieitly charaeterizing the neeessary strueture of 
the optimal eontrols. 

Define eontinuous eo-states (As, Ap, A^, Aq) sueh that at points of eontinuity of the eontrols: 


'^s — /^[(As — Xp)Gup + (As — Xz)Guz + (As — Xz)'yZ] 

Xz = -f{Z + P)+ g'{Z) + (As - Xzh/3S 

\p = -f\Z + P), ( 8 ) 

with final eo-state eonstraints: 

As(T) = Xz{T) = Xp{T) = 0. (9) 

Towards eharaeterizing properties of optimal solutions, we define the Hamiltonian as: 

Hit) := XoU\Z + P)- g{Z)) + (Ap - Xs)l3GSup 

+(Az — Xs)/3GSuz + (Xz — Xs)'y/3ZS. (10) 

Pontryagin’s Maximum Principle [38, p.l82] states that any optimal control vector u* must 
satisfy the following necessary conditions: 

(-^5; -^o) 7^ 0; (11) 

'^«ew,te[o,r] di(S*, Z*, P*,u*, Xs(t), Xp(t), Xz(t), Xo,t) > 

ns*, Z*, P*,u, Xs(t), Xp(t), Xz(t), Ao, t). (12) 

But if Ao = 0, (As(T), Ap(T), A^(T), Aq) = (), a contradiction, so Aq = 1. 

1) Structure of the optimal control: If we define: 

(fp = (Ap — Xs)/3GS (13a) 

= (Az - Xs)/3GS, (13b) 
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then, the Hamiltonian becomes: 


Hit) = f{Z + P) - g{Z) + (fpup + (pzuz 

+(Az - \s)ll^ZS. (14) 

The maximization of the Hamiltonian (12), added to the sum constraints for the controls (2a), 
leads to the following optimality conditions for the controls:^ 


(0,0) 

ifp < 0, 

Pz < 0 

(15a) 

(1.0) 

(pp > 0, 

Pp > Pz 

(15b) 

(0,1) 

^z > 0, 

Pz > Pp 

(15c) 

(?.?) 

^Z = ^P 

> 0 

(15d) 

(?.0) 

Pp = 0, 

V?z < 0 

(15e) 

(0,?) 

Pz = 0, 

Pp <D 

(15f) 


From (13) and the state (1) and costate (8) evolution equations and after some manipulations, 
we have:^ 


(pP = /3[Guz{(pz - g>p) + iZiipz - ^p) - GSf'iZ + P)] 

Pz = ms{gl{z)-f\z^p)) 

+ Gup{(pp - Lfz) - iSifz] (16a) 

pp - pz = -{^p - (pz){/3Guz + iPZ + PGup) 

-PGSg\Z)+^pSipz, (16b) 

2) Proof methodology outline: From here on, we will use the necessary optimality conditions 
to obtain timing conditions for phase transitions among the conditions in (15). We show that a 
time t* exists such that, for t G (P,T), we have up{t) = 1 and uzif) = 0 (§IV.D.3). If t* = 0, 
we have finished characterizing optimal controls. If not (i.e., t* > 0), we prove that a time t” 
exists such that for t G (t",P), we have up{t) = 0 and uz{t) = 1 (in §IV.D.4). Finally, we show 


^The question marks (?) denote singular controls. These can occur when the coefficient of a control variable in the augmented 
Hamiltonian (which includes the constraints) is zero over an interval, and thus the control has no effect on the Hamiltonian 
maximizing condition of the PMP. 

^rt’( _ ^9(7) p\ df(Z+P) df{Z+P) 

9 dz ^ ) •— dz ~ dP 
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that t" must be equal to zero (in §IV.D.5), leading to all possible optimal controls agreeing with 
the structure laid out in Theorem 2. 

3) Time interval leading up to T and the existence oft*: We now follow the evolution of 
Pz and ifp for a time interval leading to T in order to characterize necessary conditions for the 
optimal controls and to prove the existence of t*. From the terminal time costate conditions (9): 


Tp(T) — <pz(T) — 0, 

MT-) = -f{{Z + P){T-))^GS{T-) < 0, 

MT-) - fz{T-) = -l3GS{T-)g'{Z{T-)) < 0. 

Therefore, <pp{f) > max{(pz{t),t)} for some interval leading up to T due to the continuity of 
the states and costates and using the definition of a left derivative. Let {t*,T) be the largest 
interval over which this holds for t G {t*,T) for some t* < T, leading to the fact that for all 
such t, up{f) = 1 and uz{f) = 0 due to (15b). 

For t G (f*,T), (16) becomes: 

= -(3GSf\Z + P) + -il3Z{pz - Tp) (IVa) 

fz = f^GSigfZ) - f{Z + P)) + /3G{pp - pz) - if^Spz (17b) 

Pp- Pz = ll^Spz - {pp - pz){'y/3Z + /3G)-/3GSg'{Z). (17c) 

Recall that pp{t) > 0 for t e {t*,T), so due to continuity, we either have pp{t*) > 0 or 
Tp{i*) = 0- We now rule out pp{t*) = 0. If pp{t*) = 0, Rolle’s Mean Value Theorem [39, 
p. 215] applies over the interval {t*,T): as pp{t*) = pp{T) = 0 and pp is continuous and 
differentiable over this interval, there must exist r G {t*,T) such that Pp{t) = 0. However, 
from (17a), it can be seen that pp{t) < 0 for t G (P, T), a contradiction. Therefore, pp{t*) > 0. 

Thus, either P = 0 or pz{t*) = pp(t*). If P = 0, due to (I5b), we have up(t) = 1 and 

uz{t) = 0 for all t which agrees with the structure in Theorem 2, so henceforth we focus on 
the case where pz{t*) = pp{t*) > ft. 

First, we derive a property that will prove useful later on. We have .^(t) > 0 from (lb) and 

Theorem 1, and thus due to the convexity of gf) for t < t*: 

Gg'jZjt*)) ^ Gg'jZjt)) ^ 

7 “ 7 ’ 
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Next, Z{t*) can either be equal to zero or strictly positive. We first show that if Z{t*) = 0, 
the structure holds. 

If Z{t*) = 0, we have Z = ^fiSZ for t G {t*,T) as uz{t) = 0 in this interval. Consider 
Ml > 0 to be an upper-bound on the continuous 7/3S' in this interval, so we must have Z{t) < 
= 0, and therefore Z{T) = 0 due to continuity and the uniqueness of solutions 
of first-order initial value problems. Thus, as Z > 0 for t G (0,T), we must have Z = 0 over 
this interval, which from (lb) and Theorem 1 leads to uz{t) = 0 for t G (0,T) and Zq = 0. 
This also means that from (17a), 0p(f) = —(3GSf'{Z + P) < 0 in this interval, leading to 
^p{t) > ^p{T) = 0, and from (15), to up{t) = 1 over this interval. Thus, again t* = 0, agreeing 
with the structure predicted by Theorem 2. So from now on we will consider Z{t*) > 0. 

Now, we examine g'{Z{t*)) — f{{Z + P){t*)), noting that it can either be positive or strictly 
negative, and investigate both cases in turn. 

If g'{Z{t*)) - r{{Z + P)(r)) > 0, then g'{Z{t)) - f{{Z + P){t)) > 0 for all t G (r,T). 
This is because from (1), P{t) + Z{t) and Z{t) >0 over this interval, which coupled with 
the convexity of g{-) and —/(■) in their arguments gives the aforementioned result. From (17b) 
and the definition of t*, (pz > —'yl3S(fz > —M 2 (pz in this interval, with M 2 > 0 being an 
upper-bound on y/SP. Therefore, (pz{t*) < 99 z(T)e“^ 2 (t*-'r) ^ q (jjjg jq integral argument, 
which means that (pp{t*) > 0 > Lpz{t*). Note that this would contradict the starting assumption 
of this segment, which was (pp{t*) = 

Therefore, from here on we will examine the case of g'{Z{t*)) < f'{{Z + P){t*)). 

4) Time interval leading up to > 0 and the existence oft”: We now look at the evolution of 
pz and V9p for a time interval leading to t* > 0, and show that t” exists such that t for t G {t”, t*), 
we have up{f) = 0 and uz{t) = 1. Furthermore, in these cases we showed pz{P) = y^p(C), 
Z{t*) > 0, and g'{Z{t*)) < f'{{Z + P){t*)). At such a point t*, from (16a) and the continuity 
of the states and co-states: 

- fz{t*+)) = - Gg'{Z{t*))]. (19) 


Now, (19) should be positive, because if this derivative was strictly negative, the definition of 
the right-derivative would show that pz{t) > pp{t) for t in an interval starting from t*, a 
contradiction. Because from Theorem 1, S{t*) > 0, so /3S{t*)[ypzit*) — GgfZ{t*))] > 0 and: 

Gg'{Z{t*)) 


Tz{t*) > 


7 


( 20 ) 
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Now, we can see from a continuity argument on (16a) (given that = <^p(t*) > 0) that 

< 0. Thus (pz{t) > for some interval leading up to t* due to the definition of a 

left-derivative. 

, , Gg'(Z(t)) 

From (16a), (18), and (20), we must have: ^pzit) > - for t in some interval leading 

7 

up to t*. Let {f, t*) be the maximal such interval. In this interval, from (16b), ipp — ^pz > —{pp — 
Pz)i'yl3Z + (5G) > — (fz), where M 3 > 0 is an upper-bound on the continuous expres¬ 

sion 7 / 3 Z- 1 -/ 3 G. So for any t in this interval, {(pp{t) — (pz{t)) < {pp{t*)= 0. 
Thus, pp{t) < pz{i) for t G As pz{G) > 0, due to the continuity of the states and 

co-states, there exists a maximal interval {t",t*) such that pz{t) > max{(/)p(t), 0}. Following 

from (15c), for t G {t",t*) we must have up{t) = 0 and uz{t) = 1. 

5) Proof that t" = 0.’ If t" = 0, the above concludes our specification of the structure, which 
agrees with Theorem 2. Thus, henceforth we assume t” > 0, and thus either pz{t") = Pp{t'') 
or pzit”) = 0 . 

For t G (16) becomes: 

Pp = (3[-GSf\Z + P) + G{pz - pp) + lZ{pz - pp)] (21a) 

Pz = ms{g\z) - f{Z + P)) - ^Spz] (21b) 

Pp — P>z = /3[yS(pz — {pp — Pz){G + 7 Z) — GSgfZ)], (21c) 

Now, for t G (t",r), g'{Z{t)) - f{{Z + P){t)) < g'{Z{t*)) - f{{Z + < 0. This is 

because Z{t) > 0 as uzit) = 1, and P{t) = 0 as up{t) = 0, so g{-) — f{-) is convex in the strictly 
increasing Z in this interval. So from (21b), pz < —'tPSpz < —M^pz with M 4 > 0 being the 
upper-bound of the continuous 'yPS, and therefore for all t G pzif) > pzit*)e~^‘^^^~'^*\ 

and therefore by continuity, pz{t") > pz{t*)e~^‘^G'-i*)_ conclude that pz{t") > 0 , 

as pz{t*) > 0. 

So for t” > 0 , we must have ppit”) = pzif'')- In this case, we have {pp{t''^) — pz{t''^)) < 0 , 
as if it is strictly positive, an integral argument will lead to a contradiction with pp{t) < pz{t) 
for t G (t", t*). Using the continuity of the states and co-states and as from Theorem 1, S{t") > 0, 
( 21 b) becomes: 


P>pit''"") - Pz{t''"") = /3S{t'')pfpz{t'') - Gg{Z{t''))] < 0 


^ Pz{t") < 


GgfZjt")) 

7 


( 22 ) 
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We know that for all t G g'{Z(t)) — f{Z + P(t)) < 0, so from (21b), (pz{t) < 

—'y/SSifz < —M^ipz < 0, where M 5 > 0 is an upper-bound on the continuous 7 / 35 '. Thus, 

^z{t") > ^z{t*)- (23) 

But (18), (22), and (23) lead to y^z{t*) < ^ which contradicts (20). 

7 

Thus t" = 0, and this concludes our specification of the structure of the optimal controls 
which conform to the structure set out in Theorem 2. ■ 

V. Simulation 

In the preceding sections, we showed that the optimal spreading controls of the malware in 
all of the described settings can be fully described by a scalar parameter t*. In this section, we 
investigate the variation of t* with respect to some system parameters and then compare the 
relative performance of the optimal spreading controls with simple heuristics (§V.A).^ In these 
studies, the main parameter of variation is 7, as a higher 7 indicates that zombies spread at 
a faster rate than infection via germination, and thus 7 represents a measure of the virility of 
the zombie malware variant. Varying 7 changes the relative contact rates internal to the model 
and thus represents different possible dynamics of a malware attack. In contrast, varying /3, the 
contact rate of germinators and susceptibles, changes the number of contacts across the board, 
which is equivalent to changing T. Thus any variation of f3 would only show how t* changes 
for a specific epidemic. Finally, we numerically investigate the fragility of the optimal control 
to network estimation errors in the adaptive defense model and to synchronization errors among 
germinators (§V.B). 

A. Structure of the optimal malware spread controls and their performance vs heuristics 

We first computed t* (the optimal switching time) as a function of the relative spread rate of 
the zombies 7 for the problems in §III.A and §III.B (with different values of halting efficacy 
tt), as well as the optimal controls, for a cost function for which both Theorem 2 and 3 apply 
(Figure 2). As 7 increases, zombies are created for a shorter period due to the rapid explosion of 

^Stealth conscious epidemics are an emerging threat, and while more data is available now than before, their very nature 
makes real spreading data hard to come by and a topic of active research, even years after the fact. Thus, our numerical studies 
are based on simulations with parameters that are justified based on their real-world implication. 
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their population later on. Furthermore, the addition of a halting eontrol and its inereased effieaey 
leads to the attaeker ereating zombies for longer, as she ean eontrol their spread (and thus their 
visibility) later on using the halting eontrol. 



Fig. 2: We eompared t* (the length of time the zombie eontrol uz was equal to one) for the optiml 
no halting and halting eontrols as the seeondary rate of spread of the zombies ( 7 ) was varied. 
Here, f3 = 2, T = 5,(S'o, Gq, Zq, Pq) = (0.99, 0.01, 0, 0), f{x) = x^'^, and g{x) = kgX = 0.7x. 

We then eompared the eost of these two optimal eontrols to that of simple heuristies: for the 
model in §III.A, Always Zombie and Always Passive represent the two most extreme polieies 
- Always Zombie sets uz{t) = 1 and up{t) = 0 for all times, while Always Passive does the 
exaet opposite. Thus, in these heuristies the germinators only ever propagate one fixed type of 
malware variant. In the Optimal Static Mixing heuristie, the attaeker ehooses a fixed ratio for uz 
and Up at all times. Our optimal eontrols are titled No Plaiting and Halting, the latter indexed 
by the value of vr (whieh represents the relative sueeess of the germinators in halting zombies). 
The effieaey of the polieies is evaluated as 7 , the relative propagation rate of the zombies is 
varied (Figure 3, whieh is presented for the same parameters as those used in Figure 2). 

The optimal eontrols perform mueh better than the heuristies, with the halting eontrol outper¬ 
forming the no-halting eontrol for by as mueh as 10% for large values of tt (where the halting 
eontrol is effieient) and 7 (where the zombie variant propagation is rapid), both faetors whieh 
penalize sub-optimal deeision-making. This vindieates the assumption that the attaeker would be 
wise to utilize the halting eontrol were it to be available. Out of the simple heuristies, optimal 
statie mixing has the maximum utility, whieh is typieally 10% below that of even the no-halting 
optimal eontrol. 
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Fig. 3: Comparison of the damage utilities aeross the optimal eontrols and heuristics for the 
parameters of Fig. 2. 


B. Fragility of the optimal damage to network estimation errors and synchronization errors in 
the germinators 

We then investigated how the optimal control would fare when the network, which is capable of 
adaptive defense (i.e., the model in §III.C), has an erroneous estimate of the fraction of zombies 
(Figure 4). The optimal attack policy is derived with the assumption that the network’s defense 
policy is based on the correct observation of the visibility of the epidemic (i.e., the fraction 
of zombies), information that is rarely available. Figure 4 shows that the optimal control is 
remarkably robust to the network’s estimation errors up, with an error of 5% even when the 
estimation error is 40%. In many cases, the performance is much better. 

Finally, we examined how synchronization errors among the germinators would affect the 
utility of the malware. One of the benefits of the malware spread models was that they assumed 
that only this small fraction of nodes, which is under the direct control of the attacker, has to 
coordinate their actions. To examine the fragility of the optimal control to this coordination, once 
the optimal policy is derived, random errors are introduced to the clocks of the germinators, and 
the resulting utilities are compared over 100 runs of the simulation (Figure 5). As can be seen, 
the damage of both the no halting (tt = 0) and halting (tt = 0.5) cases is distributed around 
the damage obtained by the calculated optimal control, and only suffers a 10-15% performance 
drop for synchronization errors of up to 30% of t* in the small number of germinators. 

Furthermore, it can be seen that the synchronized infinite-node optimal control can actually 
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Fig. 4: The network was assumed to make unbiased random estimation errors at eaeh time instant 
with the range depieted on the x-axis. The solid line shows the average differenee in damage 
relative to the optimal over 50 runs of the estimating network. Here, we used an exponential 
sigmoid /?(Z) with (3q = I, a = 100, T = 15, 7 = 1.4, Zth = 0.01, (So, Go, ^ 0 ,-Po) = 
(0.999,0.001,0,0), and f{x) =x^-^. 


perform slightly worse than the ease where there are synchronization errors on a finite number of 
nodes, even in the mean. We can explain this as follows: in the previous sections, we characterized 
the optimal solution for the problem in §III.F under the assumption that the number of nodes was 
infinite. For a finite number of nodes, even without synchronization errors, the damage sustained 
by the simulated network can be different from (and potentially less than) that computed using 
the computational optimal control framework. 

These studies lead to the conclusion that an adversary will not be deterred by the possibility 
of errors in estimation and synchronization of the malware spread, further sounding the alarm 
about the emerging trend of visibility-aware malware. 

VI. Future Directions 

In this paper we investigated the optimal controls for the SGZP model with and without 
halting with no explicit network defense (§III.A and III.B), and without halting for the case with 
adaptive network defense (§III.C). This leaves open the case of the SGZP model with halting 
and adaptive defense. Initial analytical investigations show that Theorem 3 is likely generalizable 
to this case, barring some technical issues that will be investigated in the future. In principle, 
7 can also be a variable to be optimized by the attacker in all models. Furthermore, the model 
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Fig. 5: Germinators were assumed to have unbiased random synchronization errors at each time 
instant with the range depicted on the x-axis. The lines shows average damage over 100 runs with 
unsynchronized germinators. Here, /3 = 2 , 7 = 0.5, T = 5, (S'o, Go, ^ 0 ,-Po) = (0.99,0.01,0,0), 
f{x) = x^'^, and g{x) = kgX = 0.7x, and the simulation was run for 500 nodes (i.e., 5 
germinators). 


can be extended to a botnet case where the attack is unleashed only when the damage-visibility 
trade-off is at the optimal point - the same arguments as in the paper would hold in that case, 
with the difference that the terminal time will be free. The set-up and formulation of the visibility 
problem is, to the best of our knowledge, novel, and thus leads itself to analysis both in the 
mean-field regime and in more structured settings. In particular, in the mean-field case, possible 
patching will be addressed at a later stage, as well as the dynamic game that would result from 
such a competition. 

The current work is an abstraction of practical cybersecurity problems mainly due to the 
homogeneous mixing assumption. Another possible direction is to look at the optimal control of 
such an epidemic in sub-populations with differentiating characteristics (e.g., location, contact 
rate) as a way to relax the homogeneous mixing assumption (e.g., by following the roadmap 
in [29]). Such a generalization would better model Stuxnet in particular, with the goal being 
to maximize the number of infected agents in a particular region, while minimizing the total 
number of detectable zombies. 
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Appendix A 

Proof of Theorem 3 

Proof: This proof follows the same structure as that of Theorem 2. 

As before, we define continuous co-states (A5, Ap, A^, Aq) such that at points of continuity of 
the controls: 
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— i^s — \p)/3Gup + (A 5 — \z)[PGuz + 

\z = Xog'iZ) - Xof{Z + P) + {\s - Xzh/3S 


+ {Xz — Xp)7i/3Guh 

Xp = -Xof{Z + P), (24) 

with final state constraints: 

Xs{T) = Xz{T) = Xp{T) = 0. (25) 

To characterize optimal controls, we define the Hamiltonian to be: 

n{t) =Xo{f{Z + P) - g{Z)) + {Xp - Xz)7r/3GZUH 

+ (Xz ~ Xs)[/3GSuz + yPZS] + (Ap — Xs)/3GSup. (26) 

Pontryagin’s Maximum Principle again gives the following necessary conditions for an optimal 
control vector u*: 

{Xs, Xp, Xz, Xq) 6 AoG{0, 1}, (27) 

'^new,te[o,r] , Z*, P*,u*,Xs{t),Xp{t),Xz{t),Xo,t) > 

n{S*, Z*, P*,u, Xs{t), Xp{t), Xz{t), Ao, t). (28) 


Again, if Aq = 0, {Xs{T), Xp{T), Xz{T), Aq) = 0, a contradiction, so Aq = 1. 
Now, we have: 

Xp — Xz = —g'{Z) — (A 5 — Xz)'y/3S — {Xz — Xp)7r/3Guh 
Xs-Xz = f{Z + P) - g'{Z) + {Xs - Xp)/3Gup 

+ (A 5 — Xz)PGuz + (As — Xz)'y/3{Z — S) 

— {Xz — Xp)7rl3Guh 

Xs-Xp = f'{Z + P) + (As - Xz)\liGuz + ifiZ] 

+ (As — Xp)l3Gup, 
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1) Structure of the optimal control: If we define: 


<Pp = (Ap — 
= (Az — 
‘fh = (Ap — 

then, the Hamiltonian becomes: 


\s)/3GS 

(29a) 

\s)/3GS 

(29b) 

Az)vr/3GZ, 

(29c) 


7 i(t) = f{Z + P) - g{Z) + ifpup + (fzuz + (fhUh 
+ (Az — Xs)'y/3ZS. 

Also notice that: 

(Ph = Tr —{ pp - ipz )- (30) 

The maximization of the Hamiltonian (28), added to the sum constraints for the controls (2a), 
leads to the following optimality conditions for the controls: 


Furthermore, 


{up,Uz) = 


'(0,0) 

Pp <0, pz <0 

(31a) 

(1.0) 

ifP >0, (pp> pz 

(31b) 

(0.1) 

>0, pz > y^p 

(31c) 

(?,?) 

‘Pz = Pp > 0 

(3 Id) 

(?,0) 

99p = 0, Pz 

(31e) 


Pz = 0, Pp < 0 

(31f) 

(pp{t) > 0 

^ Up{f) + Uz{f) = 1, 

(32) 


as if that is not true, we can increase 'H(t) by adding to either up{t) or uz{t), a contradiction 
with the Hamiltonian maximization condition of the Maximum Principle (28). Also, 

0 < 0 (33a) 


Uh = { I (Ph> 0 

? ^h = 0 


(33b) 

(33c) 
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. Using (30), we can rewrite the above as: 

"0 < 9?^ & Z(t) > 0 (34a) 

Uh = < 1 Lfp > ifz Z{t) > 0 (34b) 

(fp = cpz or Z{t) =0 (34c) 

. From (29) and the state and costate evolution equations and after trite manipulation, we have: 

(fP = -(3GSf\Z + P) + /3Guz{(pz - 

+ 'y/3Z{(pz - (pp) (35a) 

ipz = f3GS{kg - f\Z + P)) - ^l3S^z 

+ l3G{up - TTUh) {(pp - (pz) (35b) 

^p-^z = -{^p - ^z){(3Guz + 'yf3Z + l3Gup - /3Guh) 

— jSGSkg + '-^l3Sipz (35c) 

iph = -n/3GZkg + nl3Guz{<pp - p>z) + nj/SZipp. (35d) 


From here on, the proof follows the same outline laid out in §IV.D.2 (in terms of finding t* and 
t” and proving t” = 0); however, the algebraic expressions for (pz, 0p are different and pihit) 
is introduced in the dynamics, necessitating the use of different and context-specific analytical 
arguments. 

2) Time interval leading up to T and the existence oft*: We follow the evolution of pz, 
p>P, and p>h for a time interval leading to T and prove the existence of t* such that we have 
up{t) = 1, uzit) = 0, and, if Z{T) > 0, Uhif) = 1 for all t e {t*,T) (otherwise, Uh can be 
arbitrary over this interval). From the terminal time costate conditions (25): 


<Pp{T) = ipziT) = (phiT) = 0, (36a) 

0p(T-) = -f'fZ + P){T-))(3GS{T-) < 0, (36b) 

0p(T-) - fz{T-) = -/3GS{T-)kg < 0, (36c) 

(ph{T-) = -TTf3GZ{T-)kg < 0. (36d) 


Now, we may either have Z(T) = 0 or Z{T) >0 due to Theorem 1. 

We start by considering the case where Z{T) = 0. From (3b) we have Z > Z{'y/3S—7r/3Guh) > 
MqZ for t G [0,T], where Mq > 0 is an upper-bound on the over the whole interval. 
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Therefore, < Z(T) = 0. Thus we must have Z{t) = 0 for all t G [0,T]. This 

means that Z{t) = f3GSuz = 0 over this interval, which from Theorem 1 leads to uz{t) = 0 for 
all t G [0,T]. Furthermore, as Z{t) is never positive, Uh{t) will have no effect on the dynamics 
of the system, and can thus be arbitrary. Finally, (35a) and (36a) tell us that (pp{T) = 0 and 
0p(f) = —/3GSf'{P) < 0 over this interval, which leads to (pp{t) > 0 for t G [0,T) due to 
continuity of the states and co-states and the differentiability of (pp{t) using an integral argument. 
This, along with Uzit) = 0 for all t G [0,T] and (32) leads to up{t) = 1 for all t G [0,T) (and 
therefore t* = 0). So in sum, for all t G [0,T), up{t) = 1, uz{t) = 0, with Uh{t) taking any 
arbitrary value. This agrees with the structure set forth in Theorem 3. 

Henceforth, we examine the case where Z{T) > 0. From (36a) and (36c), as before, (pp(t) > 
max{ipz{t), 0} for some interval leading up to T due to the continuity of the states and costates 
and using the definition of a left derivative. Let {t*,T) be the largest interval over which this 
holds for t G {t*,T) for some t* < T, leading to the fact that for all such t, upit) = 1 and 
Uzit) = 0 due to (31b). 

We now prove that for t G [t*,T], Z(t) > 0. If Z(r) = 0 at any r G it*,T), as uzit) = 0 
in this interval and from (3b) we will have Z = Zi'y{3S—7T(3Guh) < MjZ for t G [r, T] and 
for some M 7 > 0 which is an upper-bound to 7 / 35 '. This leads to Z(t) < = 0, 

or Zit) =0 for all t G [r, T] and especially 2'(T) = 0 which is a contradiction. The same 
reasoning also applies tot = t* due to continuity. So for t G [t*,T], Zit) > 0. Thus, from (34b) 
and the definition of t*, we have Uhit) = 1 for all t G (t*,T). 

So if t* = 0, we have upit) = 1, uzit) = 0, and Uhit) = 1 for all t G [0,T), which agrees 
with Theorem 3. Now we consider t* > 0. 

3) Time interval leading up to t* > and the existence oft": We now look at the evolution of 
ipz, ph for a time interval leading to t* > 0, and show t" exists such that for t G (t", t*) 

we must have upit) = 0, Uhif) = 0, and uzit) = 1. For t G (f*,T), and after replacing optimal 
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controls, (35) becomes: 

0p = -(3GSf\Z + P) + - ^p) (37a) 

= f3GS{kg - f\Z + P)) + (3G{1 - 'k){^p - ifz) 

- ll^Sifz (37b) 

^p-(pz = -{^p - (pz){l(^Z + (3G{1 - tt)) - j3GSkg 

+ 'y/3S(pz, (37c) 

^h = nZ{^l3^p-l3Gkg). {31 A) 

It can be seen that ifp{t) < 0 for t e {t*,T) (as (pp{t) > (fzit) and f'{Z{t) + Pit)) > 0 in this 
interval). This, coupled with ^fpiT) = 0 ((36a)) leads to ^pit*) > 0 due to continuity and an 
integral argument. Thus, we must have (fzit*) = ^pit*) > 0 for t* > 0. 

For t G {t*,T): 

Z + P = /3GS + 'y/lZS > 0. (38) 

Now, if kg — f'iiZ + P){t*)) > 0, then kg — /'{{Z + P)(f)) > 0 for all t G (P,T) due to the 
convexity of kg — /{■) in its argument and as Z + P is strictly increasing in this interval (from 
(38)). From (37b), (pz > —'fPSipz > —M^cpz for all t G (P,T), with Mg being an upper-bound 
on y/SS*. Therefore, (fzP*) < = 0 due to an integral argument, which means 

that (fpit*) > 0 > (fzit*). This contradicts the starting assumption of this argument, which was 

ipp{t*) = ^zit*). 

Therefore, from here on we will consider kg < /'((Z + P)(t*)). At such a point t*, from 
(37b) and the continuity of the states and co-states: 

(0p(P+) - 0z(P+)) = (5Sif)Pi^zit*) - Gkg]. (39) 

Now, (39) should be positive, because if this derivative was strictly negative, the definition 
of the right-derivative would show that 93z(t) > ‘{^pp) for t in an interval starting from t*, a 
contradiction with the definition of t*. So, as S'(t*) > 0 from Theorem 1: 

Gk 

l3Sit*)Pi^zit*) - Gkg] > 0 ^ i^ziP) > (40) 

Now, we can see from a continuity argument on (37b) (given that (fzp*) = V^p(P) > 0) that 
< 0. Thus (fzit) > ^zit*) > 0 for some interval leading up to t* due to the definition 
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of a left-derivative. Thus, from (40) we must have: (pz{t) > —- (and therefore also (pzit) > 0) 

7 

for t in some interval leading up to t*. Let be the maximal such interval. In this interval, 

from (37c), we have 

> —{^p — + /)G(1 — tt)) > —Mq{(Pp — (fz), 

where Mg > 0 is an upper-bound on the continuous expression 'j/^Z + f3G{l — tt). So for any t 
in this interval, {(pp{t) — (pz{i)) < {^p{i*) — = q_ 

Thus, ^p{t) < ^pz{i) for t G As ^zit*) > 0, due to the continuity of the states and 

co-states, there exists a maximal interval such that ^zit) > max{v?p(t), 0}. Following 

from (31c) , for t G we must have up(t) = 0 and uz{t) = 1. 

As (fzit) > from (34a) and (34c) we have Z{t)uh{t) = 0 for t G This leads 

to Z{t) >0 in this interval (from (3b)), which combined with Theorem 1 leads to Z{t) > 0 in 
this interval. Therefore, from (34a) we can also conclude that in this interval, Uh{t) = 0. 

4) Proof of t" = 0.' If t” = 0, this concludes our specification of the structure, which agrees 
with Theorem 3. Thus, henceforth we consider the case where t” > 0, and thus either (pz{t'') = 
(ppf”) or ipz{t'') = 0. 

For t G (35) becomes: 

ipP = -/3GSf{Z + P)+ /3G{ipz - ‘Pp) + ll^Z{ipz - Pp) 
pz = /3GS{kg - f{Z + P)) - -fl3Spz (41a) 

Pp-Pz = -{pp - Pz){l3G + 'y/3Z) - /3GSkg + 'jPSpz (41b) 

Ph = -7i/3GZkg + n/3G{pp - pz) + nj/3Zpp, 

Now, for t G 

kg - fiiZ + P)it)) <kg- fiiZ + P)it*)) < 0 (42) 

as kg — /(■) is convex and in this interval and P{t) + Z{t) = Z{t) = fiGS + 'y/3ZS > 0 
as uz{t) = 1, and up{t) = Uh{t) = 0. So from (41a), pz < —''^l3Spz < —Mi^pz with 
Mio > 0 being the upper-bound of the continuous 7/35', and therefore for all t G 
Pzif) > ■ As Pzit*) > 0, Pzit) is bounded away from zero, which leads to 

Pzit”) > 0 due to continuity. 
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So we must have 93p(t") = Lpz{t''). In this case, from (41b) we have (0p(t"+)—< 0, 
as if it is strictly positive, an integral argument will lead to a contradiction with 99p(t) < 
for t G {t", t*). Using the continuity of the states and co-states, as well as the fact that S{t”) > 0 
from Theorem 1, (21b) becomes: 0p(t"+) — 0^(t"+) = f3S{t")['y(pz{t") — Gkg] < 0 and so: 

(43) 

7 

From (42) and (41a), ipz < —if^Sipz < —Mi^ipz < 0. So, 

(pzit”) > ipzit*). (44) 

But (40) and (43) lead to (pz{t'') < (pz{t*), which contradicts (44). 

Thus t" = 0, and this concludes our specification of the structure of the optimal controls 
which conform to the structure set out in Theorem 3. ■ 

Appendix B 

Proof of Theorem 2 eor adaptive deeense model 

We first provide a general framework (akin to the one presented for Theorem 2), and then we 
differentiate the analysis based on the type of adaptive defense used by the network: Constant 
/3(Z) in §Appendix B.B, affine /3(Z) in §Appendix B.C, and sigmoid fi{Z) in §Appendix B.D 
As before, define the continuous co-states (A^, Ap, A^, Aq) such that at points of continuity of 
the controls: 

As = l3{Z)[{\s — \p)Gup + (As — \z){Guz + 7 -Z’)] 

Az = -\of\Z + P) + (As - \z)ll^{Z)S 

+ 13'{Z) [(As - \p)GSup + (As - \z)GSuz 
+ (As — ^z)'yZS] 

Xp = -Xof{Z + P), (45) 

with final co-state constraints: 

As(T) = Xz{T) = Ap(T) = 0. (46) 

To characterize optimal controls, we define the Hamiltonian: 

n{t) := Xof{Z + P) + (Ap - Xs)l3{Z)GSup 
+{Xz - Xs)l3{Z)GSuz + (Az - Xs)-il3{Z)ZS (47) 


35 



Pontryagin’s Maximum Principle [38, p.l82] gives us the following necessary conditions for 
optimality for an optimal control vector u*: 

{^s-, ^p-, ^o) ^ AoG{0,1}, (48) 

'^ueu ,te[o,T] Z\P\u\ Xsit), Ap(t), Xzit), Ao, t) > 

PiS*, Z\ P*,u, Xs{t), Xp{t), Xz{t), Ao, t). (49) 

But if Ao = 0, (As'(T), Ap(T), Az(T), Ao) = 0, a contradiction, so Aq = 1. 

A. General structure of the optimal control 
If we define: 

= (Ap - Xs)/3{Z)GS (50a) 

= (Az - Xs)/3{Z)GS, (50b) 

then, the Hamiltonian becomes: 

nit) = fiz + P)+ ifpup + ifzuz + (Az - Xs)il3iZ)ZS. 

The maximization of the Hamiltonian (49), added to the sum constraints for the controls (2a), 
leads to (15) as the optimality conditions for the controls: 

(fzit) > 0 or (fpit) > 0 ^ upit) + uzit) = 1, (51) 

as if that is not true, we can add to the instantaneous value of 'H(t) by adding to either Mp(t) 
or Uzit), a contradiction with the Hamiltonian maximization condition (49). 

From (50) and the state (5) and costate (45) evolution equations and after some manipulation, 
we have: 

fp = -/3iZ)GSfiZ + P)+ l3'iZ)Sipp[Guz + iZ] 

-ipp-pz)^iZ)[Guz + iZ] (52a) 

fz = -f3iZ)GSfiZ + P)- <ppGupl3'iZ)S 

-ipzPiZ)iS + ipp - pz)(iiZ)Gup, (52b) 

fp — fz = —— ^z)l^iZ)[Giuz + Up) + 7(Z + S)] 

+ippS['y/3iZ) + /3'iZ)[Giuz + up) + ^Z]] (52c) 
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Again, the proof follows the outline laid out in §IV.D.2 (i.e., proving the existenee of t* and t’, 
whieh are, however, defined differently, and proving = 0 for t* > 0), with the differenee that 
the algebraie expressions for (pz and p>p, and therefore all subsequent analytieal arguments, will 
ehange. 

1) Time interval leading up to T and the existence oft*: We follow the evolution oi pz and 
99p for a time interval leading to T and prove the existenee of t* sueh that we have up{t) = 1, 
and uz{f) = 0 for all t G {t*,T). 

From the terminal time eostate eonditions (46) and their direetional derivatives (52), we have: 

(pp{T) = (fziT) = 0, (53a) 

fp{T-) = fz{T-) = -/3{Z)GSf{Z + P) < 0. (53b) 

So, due to eontinuity of the states and eo-states, there is an interval leading up to T, over whieh 
we have pp{t) > 0 and (fzit) > 0. Let (P,T) be the maximal length interval with this property. 
For t G (t*,T), equation (51) leads to 

Uz{t) + Up{t) = 1. (54) 

Now, for t G (t*,T), (52e) becomes: 

fp{t) - (pz{t) = - {(pp - ipz)l3{Z)[G + 7(Z + S)] 

+ ppS[y/3{Z) + /3'{Z)[G + ^Z]] (55) 

The rest of the analysis depends on the 13{Z) function - we present different arguments for 
/)(Z)’s that are constant, affine, and sigmoid (§Appendices B.B, B.C, and B.D, respectively). 

For the affine case (§B.C), the analysis needs to be broken down into different cases according 

B G 

to the value of Z(T) in relation to the constant -] when I3{Z) is a sigmoid (§B.D), 

^ a 7 

we use different analytical arguments to prove the result depending on whether — 

^G — aZ{T)) + 1 is less than, equal to, or greater than zero. For the simple case of constant 
I3{Z) (§B.B), no such conditional arguments are needed. 

B. Constant /3{Z) 

Assume /3(Z) = /3f In this case, there is no penalty for creating zombies, and we expect 
zombies to be created for the whole time period. Then for t G {t*,T), (55) becomes:(0p — 


*Note that this is a case of the model in §IV.C with g = 0. 
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0z)(t) = ^pS'-i(3 - {(fp - 93 z)/ 3 [G + 7 (Z + S)] >-{(pp- (pz)Mii, for all t e {t*,T) and for 
some Mil > 0 that is an upper-bound for l3{G + ''y{Z + S)), as ipp{t)S{t)'y/3 > 0 in this interval. 
Therefore, for t G (pp{t) — (pz{t) < [(pp{T) — = 0 (from (53a)), and 

thus (pp{t) < ^zit) for t G 

Due to the eontinuity of the states and co-states and from the definition of t*, there exists an 
interval with t' < t* such that > V^p and ^pz > 0. These conditions, coupled with 

(15c) lead to up{t) = 0 and uzif) = 1 for all t G (t',T). 

We now prove t' = 0. If this does not hold, either <pz{t') = p>p{t') or p:>z{t') = 0 for some 
t' > 0 due to continuity of the states and co-states. 

Since up{t) = 0 for f G (f', T), (52b) becomes: 0z(t) = — I3{Z)GSf {Z + P) —(pzl3{Z)'^S < 
0, which leads to p>z{i') > '-Pz{T) = 0. Thus, p>z{i') cannot be equal to zero. 

If p>zit') = V^p(fO’ thsii from (52c), /3'{Z) = 0 for constant (3{Z), and the continuity of 
the states and co-states: (0p — (pz){t'~^) = (pp{t')S{t')'y/3 = (pz{t')S{t')'y/3 > 0, leading to the 
existence of an interval over which (pp{t) > (pz{t), a contradiction with the definition of 

t'. 

Thus, t' = 0 and uz{t) = 1 and up{t) = 0 for all t, which agrees with the statement of 
Theorem 2 and our intuition that zombies will be created for the entire period. □ 

C. Affine (3{Z) 

Assume I3{Z) = —aZ + /3max, with 0 < a < /3max (as /3max is an upperbound on this (i{Z) 
and I3{Z) > 0). Then, for t G (55) becomes: 

ipp{t) - ipzit) = -aLppSYl{2Z - ^^) -f G] 

-{^p - ^z){-aZ + + 7 (^ + S)] (56) 

Now we break down the situations that can arise based on the value of Z{T) with respect to 
the fixed - ^ 1 ; 

2 L a 7-1 

B G 

1) Z{T) < -]• Note that for this case, we must have — 7 ] > 0 due to 

Theorem 1. 

We first consider the sub-case where Z{T) = — 7 ] = 0. Here, we must have Z(t) = 0 

for all t as Z{t) >0 for all t and as states are continuous. The only way for Z{t) = 0 for 
all t is for us to have Zq = 0 and uz{t) = 0 for all t < T (due to Theorem 1). This leads to 
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(52a) becoming 0p(t) = — /3{0)G S (t) f' {P(t)) < 0 for all t <T, and thus (fp{t) > 0. This fact, 
combined with uz{t) = 0 for all t and (15b) leads to up{t) = 1 for all t (i.e., = 0 in the 

statement of Theorem 2). 

Otherwise, we either have (i) Z(T) = \ — ^] > 0 or (ii) Z(T) < — ^]. 

(i) In this case, from (5) (for which 13{Z) > 0 and G > 0), Theorem 1 (which specifies 

S{T) > 0), and continuity of the states, we have Z{T~) > 0. Thus Z{t) < for some 

Therefore, as Z{t) >0 for all t, so Z{t) < for all t <T. 

(ii) Since Z > 0 from (5) and Theorem 1, in this case we also have Z{t) < for 

all t < T. 

Therefore for both (i) and (ii), 7/3max — 2'jaZ{t) — Ga > 0 for all t <T. 

From (56) and for all t G {t*, T): Lpp{t)—ipz{t) > —{ipp — ipz)f3{Z)[G + 'j{Z + S)] > —{(fp — 
(pz)Mi 2 , for some M 12 > 0 which is an upper-bound to the continuous (3{Z)[G + '^{Z + S')] 
over this interval. Therefore, for t G (pp{t) — (pz{t) < [(pp{T) — = 0, 

and thus (pp{t) < ^zit) for t G 

Due to the continuity of the states and co-states and because for t G {t*,T), (pz{t) > 0, there 
exists an interval {f ,T), with t' < t* such that both ^zit) > V^p(f) and (fzit) > 0. These 
conditions, coupled with (15c) lead to up{t) = 0 and Uzit) = 1 for all t G 

We now prove t' = 0. If this does not hold, either ipz{t') = 0 or (pz{t') = fo^' some 

> 0 due to continuity of the states and co-states. 

For t G (t',T) (52b) becomes, (pzit) = —l3{Z)GSf'{Z + P) — (pz/3{Z)'yS < 0, which leads 
to (pz{t') > (pz{T) = 0. 

So we must have (pz{t') = for > 0- From (56) and the continuity of the states 

and co-states, (0p - 0z)(t'+) = ipp{t')S{t') [y/^max - 2'^aZ{t') - Ga] = ipz{t')S{t') [y/^max - 
2'yaZ{t') — Ga] > 0, leading to the existence of an interval over which (pp{t) > (fzit), 

a contradiction with the definition of t'. 

Thus, = 0 and uz{t) = 1 and up{t) = 0 for all t, which agrees with the statement of 

Theorem 2. □ 

3 G 

2) Z{T) > f[ -].• Due to the continuity of the states, Z{t) > \[^^^ — ^] fort G (ti,T) 

for some p. Recall that for t G (t*,T), (pp{t) > 0. Thus, for t G (t 2 , T), where t 2 = max{t*, ti} 
and with M 12 again defined as the upper-bound to the continuous [3{Z)[G + y(Z -f S')], (56) 
leads to: 0p(t) — 0z(t) < —{‘^p — (pz)(3{Z)[G + y(Z -f S)] < —{(pp — ipz)Mi 2 - Therefore, in 
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this interval, (pp{t) — (pz{t) > [^p{T) — ^ (pp{t) > ‘^z{t) and 

(pp(t) > 0 for t e (t 2 ,T). 

Now, due to the continuity of the states and co-states, define (f 3 ,T) to be the maximal length 
interval over which ^p{t) > max{(/) 2 (t),0}. Note that for t G (f 3 ,T) we have uz{t) = 0 and 
up{t) = 1 due to (15b). 

Due to continuity of the states and co-states, either = 0, in which case uz{t) = 0 and 
up{i) = 1 for all t (agreeing with the structure of Theorem 2), or we have a ts > 0 such that 

^pih) = 0 or ipp{ts) = (pzih) > 0 . 

From (52a), Theorem 1, and from the definition of for t G (t 3 ,T) we have, 0p = 
—I3{Z)GSf {Z + P) — {(fp — Lpz)l3{Z)'jZ — aSLfp'jZ < —aScpp'jZ < —Mi^cpp, for some 
Mi 3 > 0 that is an upper-bound to the continuous aiS'jZ over this interval. Thus, (pp{G) > 
= 0. So for G > 0, we must have (ppit^) = (fzits) > 0. From the continuity 
of the states and co-states, there must exist an interval leading up to fs such that ^pz{t) > 0 and 
(pp{t) > 0. Let (t 4 , ts) be the maximal-length interval with such a property. Notice that (51) also 

applies, leading to up{t) + uz{t) = 1 for t G (^ 4 ,^ 3 ). 

Furthermore, also from continuity, (56) becomes: 

{ipp - tpz){tt) = -a‘pp{tz)S{t:i)[-i{2Z{t^) - + G] (57) 

But if 0 p(t 3 ) — (pz{tt) < 0’ to continuity and the definition of the derivative, we 

must have an interval starting from where (pz{t) > ^p{t), which contradicts the definition 
of ^3 (which stated that over an interval starting at G, (pp{t) > max{(/)p(t), 0}). So we must 
have ^p{t^) — ^z{tt) > 0. From (57) this is equivalent to [y{2Z{t3) — -i- (7] <0, or 

Following the same set of arguments as presented in §B.C. 1 for the case of Z(T) < f 
and retracing them for Z{t 3 ) < (with replacing T in all arguments) shows that 

the structure postulated in Theorem 2 holds. 

Thus, all possible state and co-state trajectories lead to the structure postulated in Theorem 

2 . □ 


D. Sigmoid l3{Z) 
Assume /3z = 


l^o 


-, with 0 < Zth < I being a fixed threshold and a > 0 denoting 


]_ -|- pOL{Z— Zth) 

the sharpness of the cut-off. This simulates a threshold-like detection of zombies by a network 
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administrator. In this case, (52c) becomes: 


^p-(pz = -{(fp - ‘fz)l3{Z)[G{uz + Up) + 'j{Z + S')] 

- ^G{u^ + Up) - aZ) + l] 

(1 -|- g“(-^~-^t/i))2 

Define: '^{Z,uz + up) := — ^G{uz + up) — aZ) + 1. Then (58) becomes: 


^p- = -{^p - ‘^z)f3{Z)[G{uz + Up) + 7(Z + S')] 

PojippS 


+ 


-\I/(Z, uz + Up) 


(59) 


(1 + e “(^“'^ i '>))2 

Now, for possible intervals where + wp is a constant c G [0,1], ^(Z, c) is a function of one 
variable (Z). We can see that at points of continuity of the controls and in intervals where it is 
defined, ^{Z, c) is also continuous and differentiable. Furthermore, we can see that at points of 
continuity of the controls in these intervals, we have: 


d^{Z, c) 






(—c + Z) < 0 
7 


(60) 


Now we break down the situations that can arise based on the value of \1/(Z(T), 1): 

1) \h(Z(T), 1) > 0.' From Z > 0 ((5) and Theorem 1) and the continuity of the states, we 
have Z{t) < Z{T) for all t. Now for t G {t*,T), as the sum of the controls is constant and equal 
to one due to (54), we will have \E'(Z(t), 1) > 4'(Z(T), 1) > 0 due to (60). Thus from (59) and 
for all t G {t*,T) at which the controls are continuous: (pp{t) — > —(v^p — ^z)l3{Z)[G + 

7 (Z + S')] > —{(fp — (pz)Mi 4 , for some M 14 > 0 which is an upper-bound to the continuous 
(3{Z)[G+-i{Z + S)]. Therefore, fort G (r,T), ifp{t)-ifz{t) < [ipp{T)-ifz{T)]e-^^^^^-^^ = 0, 
and thus (fp{t) < ^pzif) for t G {t*,T). 

Due to the continuity of the states and co-states and from the definition of t*, there exists 
an interval {t',T), with t' < t* such that ^fzit) > ^p{t) and (fzit) > 0. These conditions, 
coupled with (15c) lead to up{t) = 0 and uz{t) = 1 for all t G {t',T), with the corollary that 

Up{t) + Uz{t) = 1. 

We now prove t' = 0. If this does not hold, either ipz{t') = 0 or (fzit') = ‘^p{t') > 0 for 
t' > 0 due to continuity of the states and co-states. 

For t G {t',T), as up{t) = 0, (52b) becomes: ^z(t) = —l3{Z)GSf'{Z + P) — (pzl3{Z)'yS < 
0, as each term in the right hand side is strictly positive in the interval. Now, if we have 
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‘fzit') = 0, from this time-derivative and continuity of the states and co-states we must have 
> ^z{T) = 0. Thus, = 0 is ruled out. 


On the other hand, if (pz{t') 
and co-states: (0p — (pz) = 


= ipp(t') > 0, then from (59) and the continuity of the states 
) x]/(^(^^)^ 1) > 0, leading to the existence of 

(1 “ 1 “ 


an interval over which (pp{t) > p>z{t), a contradiction with the definition of t'. 


Thus, t' = 0 and uzit) = 1 and up{t) = 0 for all t, which agrees with the statement of 


Theorem 2. 


2) 'h(Z(T),l) = 0 and Z{T) > 0.- We have Z{T~) > 0 (from (5), Theorem 1, and continuity) 
which leads to Z{t) < Z{T) for an interval leading up to t. As Z > 0, we can extend Z{t) < 
Z{T) to all t. Now for t G from (54), we will have '^{Z{t), 1) > \h(Z(T), 1) = 0 due 

to (60). We now prove = 0 and uz{t) = 1 and up{t) = 0 for all t. 

From (54), (59), for all t G it*,T) (over which (fpit) > 0): 0p(t) — 0z(t) > —{'Pp — 
Pz)[^iZ)[G + 7 (Z -f S')] > —{pp — Pz)Mi 2 for some M 12 > 0 which is an upper-bound to the 
continuous (3iZ)[G + 7 (Z -f S')] over this interval. Therefore, for t G (t*,T), ppit) — pzit) < 
[ppiT) — 99z(T)]e“^i2‘^*“'^^ = 0, and thus pp{t) < pzit) for t G {t*,T). 

Due to the continuity of the states and co-states and because for t G (t*,T), pzit) > 0, there 
exists an interval (t',T), with f < t* such that both pzit) > ppit) and pzit) > 0. These 
conditions, coupled with (15c) lead to Mp(t) = 0 and uzit) = 1 for all t G (t',T). 

We now prove = 0. If this does not hold, either (i) pzit') = 0 or (ii) pzit') = Ppit') for 
some > 0 due to continuity of the states and co-states. 

For t G (t',T) (52b) becomes: pzit) = —l3iZ)GSf'iZ + P) — pzl3iZ)'^S < 0, which leads 
to Pzit') > PziT) = 0. 


So for > 0 we must have pzit') = ppit'). From (59) and the continuity of the states and co- 

(rn -rn = l^o^^pit')Sit') ^ ^ Pojpzit')Sit') . 

states, ((^p pz)it ) ), 1) ^ ), 1) > 0, 

leading to the existence of an interval it',t") over which ppit) > pzit), a contradiction with 


the definition of t'. 


Thus, t' = 0 and uzit) = 1 and Mp(t) = 0 for all t, which agrees with the statement of 
Theorem 2. □ 


3) 4'(Z(T), 1) = 0 and 2’(T) = 0.’ We must have Z(t) = 0 for all t as Z > 0 and as states 
are continuous. The only way for Z(t) = 0 for all t is for us to have Zq = 0 and uzit) = 0 for 
all t < T (due to Theorem 1). This leads to (52a) becoming ppit) = —/3(0)GS'(f)/'(P(t)) < 0 
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for all t <T, and thus (pp{t) > 0. This fact, combined with uz{t) = 0 for all t and (15b) leads 
to up(t) = 1 for all t. 

4) \h(Z(T),l) < 0; Due to the continuity of the states, < 0 for t G (ti,T) for 

some ti. Thus, (59) leads to 0p(t)-^^(t) < -{(pp-(pz)/3{Z)[G+'-r{Z+S)] < -{(pp-(pz)Mi 2 , 
for t G {t 2 :T), where t 2 = max{t*,ti} and with M 12 defined as before (an upper-bound to the 
continuous l3{Z)[G + 7(2' + S')] over this interval). Therefore, in this interval, ^p{t) — (pz{t) > 
[(pp{T) — (/)^(T)]e“^i 2 7-'r) ^ (pp{t) > (pz{t) and (pp{t) > 0 for t G {t 2 ,T). 

Now, due to the continuity of the states and co-states, define (f 3 ,T) to be the maximal length 
interval over which ^p{t) > (pz{t) and (pp{t) > 0. Note that for t G {h-,T) we have (due to 
(15b)) uz{t) = 0 and up{t) = 1. 

Due to continuity of the states and co-states, either = 0, in which case uz{t) = 0 and 
up{t) = 1 for all t, or we have a fs > 0 such that (i) ipp{ts) = 0 or (ii) (fpiG) = (fzits) > 0 . 
From (52a), Theorem 1, and from the definition of for t G (t 3 ,T) we have: ipp = 

-P{Z)GSf\Z + P)-[^P-^z)m,Z-j^^^^^ < - < 

—Mi^ipp, for some M 15 > 0 that is an upper-bound to the continuous ^ a{z-Zth)Y ' 

(Pp{h) > 99p(T)e"^15(‘3-T) ^ Q_ 

So for ^3 > 0 we must have (fpit^) = (pzih) > 0. From the continuity of the states and 
co-states, there must exist an interval leading up to ts such that (fzit) > 0 and (pp{t) > 0 . 
Let (t 4 ,t 3 ) be the maximal-length interval with such a property. Notice that (51) also applies, 
leading to up{t) + uz{t) = 1 for t G (^ 4 ,^ 3 ). 

Furthermore, also from continuity, (59) becomes: 

Mtt) - 1 ) <«> 

But if 0 p(t 3 ) — (pz{tt) < 0’ then due to continuity and the definition of the derivative, we must 
have an interval starting from where ^z{t) > (pp{t), which contradicts the definition of ^ 3 . So 
we must have p>p{t^) — p>zitt) — 0- From (57) this is equivalent to '^{Z{ts), 1) > 0. Following 
the same arguments presented in §B.D.l, §B.D.2, and §B.D.3 for the case of \1'(Z(T), 1) > 0 
and retracing them for \h(Z(t 3 ), 1) > 0 (with G replacing T) shows Theorem 2’s structure holds. 
Thus, Theorem 2 holds for all possible trajectories. 
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